HTTP policy control (concurrency, etc) - #429

Summary

Add administrator-enforced HTTP concurrency limits and workflow-local policies, with nonblocking admission so requests waiting for capacity remain durably queued rather than occupying activity slots.

Server-wide limits

Add two restart-only GUCs, both defaulting to 2:

• pg_durable.max_activity_concurrency: total concurrent activities, including SQL, HTTP, and metadata work.
• pg_durable.max_http_concurrency: combined HTTP and multipart HTTP capacity across all users, workflows, and origin databases.

Defaults preserve the previous two-activity execution capacity. Administrators can set the HTTP ceiling below total activity capacity to leave room for non-HTTP work.

Workflow-local policies

Add df.with_policy(fut, policy) to constrain HTTP concurrency within an entire workflow or a subtree:

SELECT df.start(
    df.with_policy(
        df.join(
            df.http('https://api.github.com/repos/microsoft/pg_durable', 'GET'),
            df.http('https://api.github.com/repos/microsoft/duroxide', 'GET')
        ),
        '{"http": {"max_concurrency": 1}}'::jsonb
    ),
    'repository-metadata'
);

• All enclosing scopes and the global ceiling apply together; admission reserves all required capacity atomically or reserves none.
• Inner policies cannot relax outer limits. Reapplying a policy to the same subtree root keeps the smaller limit.
• Parallel branches and loop generations share their enclosing scopes, including after restart. Separate starts and separate occurrences of reused fragments have independent local scopes.
• Policy input is strictly validated, and df.explain() displays applicable scope IDs and limits.

Execution behavior

• Admission occurs before origin routing and credential lookup. Permits cover request preparation, response-body processing, and optional sink storage.
• Capacity waiting does not consume the request timeout. Committed deferrals do not record activity results or spend the fetch-attempt budget; infrastructure failures can still consume attempts.
• The provider orders deferred work behind already-ready items, preventing a saturated HTTP backlog from indefinitely overtaking unrelated SQL and metadata activities.
• HTTP authorization, destination restrictions, and credential isolation remain unchanged.
• Also make runtime startup/publication I/O interruptible to fix a shutdown race found during validation.

This does not add rate limits, timeout/deadline policies, per-tenant fairness, or exactly-once/remote-quiescence guarantees. Large backlogs still add latency.

Dependencies

Uses immutable Git pins for:

• Add nonblocking activity admission control duroxide#74 (on main now) — merged admission support.
• Prevent deferred worker items from starving ready work duroxide-pg#51 — pending PostgreSQL queue-ordering fix.

These pins allow combined testing and review before upstream releases; they are temporary integration dependencies.

Upgrade and compatibility

• Global limits apply after a binary update, including on supported older extension schemas.
• df.with_policy() requires ALTER EXTENSION pg_durable UPDATE TO '0.2.10'.
• Policies use existing HTTP configuration storage: no new policy tables, executable node types, or orchestration permit operations.
• Unconfigured HTTP payloads retain their existing bytes for replay compatibility.
• Worker readiness advances to 4, published after the bundled provider migrations complete.
• Migration guidance covers index-build locking, concurrent prebuilding, and older binaries rejecting newer provider migration state on startup.